Regression with Multiple Candidate Models: Selecting or Mixing?

Model averaging provides an alternative to model selection. An algorithm ARM rooted in information theory is proposed to combine di erent regression models/methods. A simulation is conducted in the context of linear regression to compare its performance with familiar model selection criteria AIC and BIC, and also with some Bayesian model averaging (BMA) methods. The simulation suggests the following. Selection can yield a smaller risk when the random error is weak relative to the signal. However, when the random noise level gets higher, ARM produces a better or even much better estimator. That is, mixing is advantageous when there is a certain degree of uncertainty in choosing the right model. In addition, it is demonstrated that when AIC and BIC are combined, the mixed estimator automatically behaves like the better one. A comparison with bagging (Breiman (1996)) suggests that ARM does better than simply stablizing model selection estimators. In our simulation, ARM also performs better than BMA techniques based on BIC approximation. ARM is a computationally feasible way to combine models and/or non-model-based procedures. It is a convex combination of the original estimators with data-dependent weights. For the determination of the weights, the data is split into two parts. The rst one is used for estimation by each model or procedure and the accuracies of these estimators are assessed using the second half of the data. The accuracies are then used to assign the weights in a way such that a connection between function estimation and information theory ensures a desired theoretical capability of adaptation over di erent models and/or regression procedures.